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1. INTRODUCTION 

In recent years, robotics research has become proficient with a significant alteration in the 
automation sector, object handling and tacking applications. The research comforts are moving from the 
development of robots for organizing manufacturing industries to the expansion of independent movable 
robots functioning in unstructured and natural environments. These independent mobile robots are 
appropriate in huge number of complex tasks solving such as vacuuming of hazardous material, 
investigation, liberation and investigation in free surroundings wherever humans are reserved from easy 
access. Since it is predicted that this novel class of movable robots will have wide applications in 
accomplishments where human competences are needed, they have concerned the consideration of the 
investigators [1]. Mobile manipulator robot is currently an extensive shift that denotes the robots poised with 
a robotic arm straddling on a movable dais. Such system is usually categorized by a huge degree of 
redundancy and it also associates the manipulability of a static base robotic manipulator with the mobility of 
a movable dais. These kinds of systems allow the most normal tasks of robotic systems that require both 
manipulation tasks and movement. Such systems promises to provide numerous applications in diverse 
manufacturing and industrious areas as construction and mining or for public assistance [2,3]. 

The declined price of the Field Programmable Gate Arrays (FPGAs), their growing competences, 
and the likelihood to increase the performance of processing and computation of tasks with unique hardware 
features, have improved their usage in the applications associated with control and many more. The current 
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abilities of FPGAs provide a specific hardware technology, which can be engaged to implement embedded 
control systems [4]. FPGAs have previously been used with triumph in diverse systems as the growth of 
advanced degree polynomial profile generator for CNC and robotics applications [5], control of robotic 
manipulator for industrial applications [6], neural and fuzzy based controllers for certain critical 
applications[7], mechatronic and other automation systems [8], etc. FPGAs and their fractional 
reconfigurability afford supplementary possessions to control systems for processing, interface, testing, 
configuration capabilities, etc.). In this article, a visual servoing system based FPGA is proposed to 
accomplish the regulation of robotic manipulator for object tracking applications using Hardware-in-the-loop 
simulation (HIL) technique. 

When high precision and performance of position control of a robotic manipulator for object 
tracking is considered, visual feedback can be extensively used to improve the accuracy of the end effector 
for positioning the objects. Visual information, disparate the ordinary sensors yield a measurement of the tip 
position, evading approximation errors due to a coarse familiarity of the physical constraints symbolizing the 
mechanical assembly of the robotic manipulator. Visual servoing based control of robots has been growing in 
recent years [9,10]. 

The robot manipulator controller positioning for object tracking can be hardware or software 
oriented device that constantly processes the value of the feedback from the joint sensors and then actuates 
the joints of the robotic manipulator to exact any deviation from the desired value [11]. The joint positions 
and velocities are measured by joint sensors such as potentiometers, tacho generators or digital encoders. The 
HIL based simulation modelling aids, to fast prototyping of control algorithms, rather than researchers 
investing on the actual robotic manipulator[12,13]. In this present work, the control algorithms are 
implemented on Xilinx Virtex-6 FPGA board. The implementation part is tested with the HIL simulated 
model of the three link serial robotic manipulator. Figure 1 shows the overall HIL block diagram 
implementation of the proposed object tracking controller. 
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Figure 1. HIL block diagram of the overall object tracking controller 


2. DYNAMIC PERCEPTIBLITY BASED OBJECT TRACKING 

Tracking of objects based on dynamic perceptibility is supplemented by the usage of camera model 
tracking the end effector of the HIL model of robotic manipulator. Representing pas the pose of camera 
model located at the end effector and tracking the end effector of the HIL model of robotic manipulator. 
Representing pas the pose of camera model located at the end effector and e=|fx,fy,f2x,....,fjxfgy|T ER2j as 
a vector of j number of extracted feature points of the object desired to be tracked [14,15]. The velocity 
relation in the extracted feature points is represented by 


(1) 
e, =I (e)p 


Where the interaction matrix Is(e) affects the extracted feature point velocity ep based on the change 
on the pose of the camera. They can be related to the joint space velocity as, 


i (2) 
e, =J, DG =1;4.8)q 
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Where Jp(q)is the Jacobin matrix of the robotic manipulator and Ij = Ij(q, e) represents the Jacobin 
matrix mapping from joint space to the extracted feature point space. The acceleration of the extracted 
feature point is given by the second derivative of equation (2) with respect to time. 


. a (3) 
e,=l1;q+4q 


By neglecting the friction coefficients of the robotic manipulator the dynamic mode the serial rigid 
robotic manipulator of n link can be represented as, 


7 l (4) 
T=M(QG+C@.G@ + sq) 


Introducing a scalar value w for determining the ability of positing of object by therobotic 
manipulator, it can be defined as in equation (5) taking into account the joint velocities in unit form. 


w= 4/det(1 1)" (5) 


Considering the differential kinematics Ir of the robotic manipulator and relating it to the velocity 
joint velocities qand end effector velocity pwe obtain the unit form expression as 


ee 6) 
P I7] r=1 


The manipulability scalar value can be obtained as 


W, = det{Z,(M7M)"T,"] u 


By assuming the joint velocities , =0, the dynamic manipulability can be given by: 


a ore (8) 
P U,M'M"™T.)' P=0 


It is derived based on ignoring the gravity terms such that 


; (9) 
T=Mq 


The closeness of the extracted feature points measures the singularity of the features by defining the 
perceptibility wPas a scalar function defined as 


Wp = det{I.7.") ne 


It represents the kinematic evidence and the following vectors are defined toe estimate the dynamic 
perceptibility value. 


5 . (11) 
T=T-cC-8=Mq 


(12) 


~ 


g=s—-l,q=1,q 


From the equations (11) and (12) the following relation can be obtained as 
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m - (13) 
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The dynamic perceptibility wdp, can be obtained as 


wp =afdetL iM (I M>)" ] a 


A 2jx2j matrix results while multiplying the term IJM-1(IJM-1)T.This matrix holds the dynamic 
perceptibility information for each extracted visual feature points. It is feasible to represent } number of 
ellipses for each of the extracted features. Based on the movement capability of the end effector indicated by 
the ellipses it is often desirable to make the robotic manipulator move forward or backward [16]. 

The trajectory of the extracted feature is obtained by expressing reference acceleration of the 
extracted features using the proportional gain matrix Kp and derivative gain matrix KD. It can be expressed 
as: 


. (15) 
é.=é€,+k,(e,—e)+ K(e,—e) =e, +kprs+ Ky, 


Where , and are the desired position, velocity and accelerations of the extracted features of the 
object. The equation (3) and (15) can be related as: 


j ` ' ' ' ' (16) 
L, q=ea+ Kpr, +K r,—l.g 


The error in the extracted feature in the controller is reformed to force the robotic manipulator in the 
direction of largest eigen vector of each dynamic perceptibility ellipses. 


rs=rs+||rs||GU (17) 


Where rs+[rs|| represents the norm of the extracted features of the object and G is the gain matrix 
that controls the effect of the perceptibility on each visual feature. GU is the diagonal matrix where values of 
tis diagonal are unit vectors pointingthe direction of the largest eigen vector [17,18]. 

Applied torques for the joint actuators of the robotic manipulator for tracing the objects can be 
definedas 


r =W" (GMW) (b—GM |(-C- g)) (18) 


Where W represents the time dependent weight matrix and the symbol + points out the pseudo 
inverse of the matrix and the constraints of the proposed task is given by: 


a l (19) 
G(q,q,t)gq =b(q,q,t) 


Where G(g,g,t)q € R™” and b(q,q,t)q eR” By representing the task constraint of the 
relation in equation (16) in the form in equation (19) with G=I, 


P . (20) 
b =eatkpr,—l, q 


The control law concerned on the feature description of the task is set by: 


. (21) 
t=MI,+(eatkpr,tk,r,—-I gq )+ct+g 
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It is derived from equation (20) such that replacing the variables concerned on the feature task 
description [19, 20, 21, 22]. 


3. IMPLEMENTATION OF THE OBJECT TRACKING CONTROLLER WITH FPGA 

The vital considerations about the FPGA implementation of the object tracking visual servoing are 
indicated in equation (21). The pipeline of the implemented controller is shown in Figure that depicts the 
parallel architecture of the object tracking using servoing accurately. Response of the controller constrained 
to time factors is managed by the proposed architecture. 

The five stage module of the visual servoing system comprises of pipelined conceptualization of 
observing, resizing, thresholding, erosion and detection of the objects. The vision algorithms take up the 
pipeline execution stages in the FPGA. 

The latency of the processing stages are considerably reduced thereby resources for storage and 
routing of data traffic are minimized. Time chart of the pipelined servoing for object tracking for the HIL 
model of the robotic manipulator is shown in Figure 2. The scheme of functioning of the implemented 
controller for object tracking is coded in Hardware Description Language (HDL) by taking advantage of the 
pipeline stages in FPGA. 

By exploiting these capabilities the matrices IJ, M and C are computed parallel making other 
dependent ones to wait for their inputs. It is also considered by sharing the cones by optimization strategies. 
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Figure 2. The scheme of the control module implemented in the Xilinx Virtex 6 FPGA XC6VLX760 device 


In the proposed HIL simulated model that runs on the MATLAB simulink environment is triggered 
by the control inputs from the FPGA. The joints of the robotic manipulator and their torque requirement 
based on object location are fed from the FPGA to the simulink model. Thereby the position of the end- 
effector is tracked using visual servo modeling of the vision system making the coordinates of the object 
under test. The moment of the object holded in the end-effector is continuously tracked by the vision 
algorithm based on dynamic perceptibility implemented in the FPGA. 

The ellipses presented in Figure 3(a) and Figure 3(b) signify the movement competence of every 
extracted feature points in the pattern. Here a pattern that consists of three points is considered. The 
representation in Figure 3(a) shows the perceptibility ellipses and it is acquired only with data available in the 
image space, which makes the pattern to be perfectly circle. The illustration in Figure 3(b) shows the 
dynamic perceptibility ellipses instead, take into consideration the dynamics of the robotic manipulator, 
influencing the contour of the ellipses there concluding that it would be better to change the pattern up or 
down, instead of sidelong. 
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Figure 3. (a) Representation of Perceptibility ellipses. (b) Representation of Dynamic perceptibility ellipses 


4. EXPERIMENTAL RESULTS AND DISCUSSION 

This discussion presents the investigations carriedout to exemplify the performance of the visual 
servoing robotic manipulator architecture for object tracking based on FPGA. Additionally, the value of the 
dynamic perceptibility idea suggested in this paper is presented here. The trials shows that this idea affords 
facts about the system capability to track objects using a controller based on direct visual technique 
implemented in FPGA. This parameter is a useful measure to govern circumstances where the robot is nearby 
singularity. The futures of dynamic visual controller implemented in the Xilinx Virtex 6 FPGA board are 
tested with the HIL model of the three link robotic manipulator implemented in the MATLAB Simulink 
environment. Additionally, the redundancy of the HIL model of the three link robotic manipulator allows us 
present the results attained for more challenging tasks. In the experiment described the camera model 
implemented is calibrated with intrinsic parameters in certain optical centre pointing the focal length in X and 
Y directions. Owing to the restrictions of this three link robotic manipulator and taking appropriate measures 
to avoid singularities, the work taken up for experimentation is considered to be in a 2D plane matching to 
the object position for the preliminary tests. The robotic manipulator implemented has 3 dof, but to 
accomplish real environment visual servoing for object handling the robot needs to have minimum of 6 dof. 
The results part paper not addresses the challengling parts of processing tasks on the captured objects, but 
rather dwells in representing that this system is extremely robust and that it provides high degree of re- 
configurability with parallel and fast processing with the aid of FPGA. The investigations are carried out by 
first positioning the robotic manipulator at the desired coordinates in the plane. The object's under test that 
needs to be tracked is observed by storing its centre of gravity (c.o.g). For implementing the Direct Visual 
controller as represented in equation (21), the constants KP and KD are used such that its initial value are 


chosen as 
600 0 200 0 (22) 
K = ‚Kp = 
j O 1300 QO 350 


The navigation of the end-effector of robotic manipulator implemented as a HIL model, moves the 
camera model implemented with a fairly straight path, hardly with negligible oscillations, and without 
omitting the preferred position. Evolution of the norm error of the extracted features pointes 1s shown in 
Figure 4a. It can be observed that the error falls quickly after few initial iterations. Figure 4b demonstrations 
the output of the actual joint torques of the links during the placing task of the object, the torques inputs sent 
from the controller implemented in the FPGA to the motors of each robot joint. Around, after 1.5s, the 
system reaches the position there by the required torque reaches close to 0. 
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Figure 4. Progression of the (a) Norm of error in the extracted feature points. (b) Output of the system joint 


torques of the robotic manipulator. 


Figure 5 shows the value of the dynamic perceptibility obtained during the investigations. Figure 5a 
and Figure 5b represent the value of dynamic perceptibility using the direct visual servoing controller and 
dynamic perceptibility based visual servoing controller, respectively. As expected, by using the dynamic 
perceptibility system, it perceives the condition where the robotic manipulator is near to the singularity and 
the modification of the robot trajectory raises the dynamic perceptibility, avoiding the singularity 
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Figure 5.Dynamic perceptibility observed in the HIL simulation for various iteration using (a)Direct visual 
servoing controller. (b)Dynamic Perceptibility based Visual Servoing controller. 


As a final part of the implemented controller Figure 6 


Visual Servoing 


Controller ans 
Direct 28938 
(12%) 

Dynamic 60288 
Perceptibility (25%) 


Table 1. EL=RPL rank calculation process 


Slices 


5275 
(14%) 
13565 
(36%) 


DSP 
blocks 


LUT- 
RAM 
105 
(0.7%) 
285 
(1.9%) 


Execution 
time 


2.85 ^s 


3.20 ^s 


As a final part of the implemented controller Figure 6 shows the three dimensional trajectories of the 
robotic manipulator obtained from the direct visual servoing controller and dynamic perceptibility based 
visual servoing controller. It represents the modifications made by the end effector of the robotic manipulator 
using the proposed controller implemented inFPGA targeting to avoid the robot singularity. It is also evident 
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from the resulting trajectory that the singularity in the robotic manipulator is eliminated thereby provides 
better tracking of the trajectory to track the objects. 





Figure 6. Observed trajectories of three link robotic manipulator for direct visual servoing controller shown 
in solid red link and the trajectory of Dynamic Perceptibility based Visual servoing controller in blue dashed 
link 


The resource utilizations ofthe implemented controller in the FPGA are summarized in the TABLE. 
1. Xilinx Virtex6 FPGA that houses the device XC6VLX760 is used to the experimentation. The percentage 
of the resources used isalso indicated. It is evident that the implementation of the dynamic perceptibility 
based visual servoing controllerin the FPGA acquires more resources than the direct visual servoing 
controller. Also, the execution time of the proposed algorithms istabulated representingthe incurred in 
execution of the algorithm in the FPGA device. Based on the excess resources in the direct visual servoing 
controller, it also consumesaround 3.20us delay in completing the servoing task of object tracking. 


5. CONCLUSION 

A direct visual servoing controller implementation on an FPGA to accomplish the supervision of an 
object hold on by a HIL model of a three link rigid robotic manipulator using the camera model system is 
suggested in this work. The implementation of the reconfigurable hardware architecture permits to acquire 
persistent processing intervals and ideal set of tasks by performing pipeline processing of the sub tasks. 
Furthermore, the dynamic manipulability based object tracking is implemented to determine the dexterity of 
the robotic manipulator to track desired objects by utilizing the direct visual controller. The investigations are 
also performed using dynamic perceptibility based visual servoing and results are compared with the direct 
method. It is obvious and evident that the proposed dynamic perceptibility based visual servoing utilizes 
optimum resources in the hardware architecture with ideal performance and better latency. The observed 
performances are accountable to the service of object tracking and the control of the joints of the robotic 
manipulator that handles the desired object. 
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